Lia/upgrade tonic to v14 - #12
Draft
jkosh44-datadog wants to merge 27 commits into
Draft
Conversation
* Rework distributed planner * Ignore some failing tests that should not be failing * Improve default task estimator repartition * Update test snapshots * Factor out plan_annotator.rs * Add docs * Respond to PR feedback * Make TaskCountAnnotation public and part of the TaskEstimator API * Add plan annotator tests (datafusion-contrib#263) * Fix typo * Add docs
* Make TPC-DS tests use DataFusion test dataset * Delete unnecessary scripts and tweak pipelines * Add TPC-DS plan assertion tests * Answer to feedback * Do not use multithreading
* Split channel resolver in two * Simplify WorkerResolverExtension and ChannelResolverExtension * Add default builder to ArrowFlightEndpoint * Add some docs * Listen to clippy * Split get_flight_client_for_url in two * Fix conflicts * Remove unnecessary channel resolver * Improve WorkerResolver docs * Use one ChannelResolver per runtime * Improve error reporting on client connection failure * Add a from_session_builder method for constructing an InMemoryChannelResolver * Add ChannelResolver and WorkerResolver default implementations for Arcs * Add test for default channel resolver * Extend doc comment * Fix typo
…ib#269) * Split channel resolver in two * Simplify WorkerResolverExtension and ChannelResolverExtension * Add default builder to ArrowFlightEndpoint * Add some docs * Listen to clippy * Split get_flight_client_for_url in two * Fix conflicts * Remove unnecessary channel resolver * Improve WorkerResolver docs * Use one ChannelResolver per runtime * Improve error reporting on client connection failure * Add a from_session_builder method for constructing an InMemoryChannelResolver * Add ChannelResolver and WorkerResolver default implementations for Arcs * Make TPC-DS tests use DataFusion test dataset * Remove non-working in-memory option from benchmarks * Remove unnecessary utils folder * Refactor benchmark folder * Rename to prepare_tpch.rs * Adapt benchmarks for TPC-DS * Update benchmarks README.md * Fix conflicts * Use default session state builder * Update benchmarks README.md * Make gen-tpcds.sh executable
* Split channel resolver in two * Simplify WorkerResolverExtension and ChannelResolverExtension * Add default builder to ArrowFlightEndpoint * Add some docs * Listen to clippy * Split get_flight_client_for_url in two * Fix conflicts * Remove unnecessary channel resolver * Improve WorkerResolver docs * Use one ChannelResolver per runtime * Improve error reporting on client connection failure * Add a from_session_builder method for constructing an InMemoryChannelResolver * Add ChannelResolver and WorkerResolver default implementations for Arcs * Make TPC-DS tests use DataFusion test dataset * Remove non-working in-memory option from benchmarks * Remove unnecessary utils folder * Refactor benchmark folder * Rename to prepare_tpch.rs * Adapt benchmarks for TPC-DS * Update benchmarks README.md * Fix conflicts * Use default session state builder * Add clickbench correctness tests * Add clickbench plans tests * Ignore also clickbench 3 * Rename file * Add clickbench to benchmarks
* Split channel resolver in two * Simplify WorkerResolverExtension and ChannelResolverExtension * Add default builder to ArrowFlightEndpoint * Add some docs * Listen to clippy * Split get_flight_client_for_url in two * Fix conflicts * Remove unnecessary channel resolver * Improve WorkerResolver docs * Use one ChannelResolver per runtime * Improve error reporting on client connection failure * Add a from_session_builder method for constructing an InMemoryChannelResolver * Add ChannelResolver and WorkerResolver default implementations for Arcs * Make TPC-DS tests use DataFusion test dataset * Remove non-working in-memory option from benchmarks * Remove unnecessary utils folder * Refactor benchmark folder * Rename to prepare_tpch.rs * Adapt benchmarks for TPC-DS * Update benchmarks README.md * Fix conflicts * Use default session state builder * Add ChildrenIsolatorUnionExec * Add proto serde for ChildrenIsolatorUnionExec * Wire up ChildrenIsolatorUnionExec to planner * Add integration tests for distributed UNIONs * Skip query 72 in TPC-DS benchmarks * Allow setting children isolator unions * Allow passing multiple queries in benchmarks
* Split channel resolver in two * Simplify WorkerResolverExtension and ChannelResolverExtension * Add default builder to ArrowFlightEndpoint * Add some docs * Listen to clippy * Split get_flight_client_for_url in two * Fix conflicts * Remove unnecessary channel resolver * Improve WorkerResolver docs * Use one ChannelResolver per runtime * Improve error reporting on client connection failure * Add a from_session_builder method for constructing an InMemoryChannelResolver * Add ChannelResolver and WorkerResolver default implementations for Arcs * Update missing documentation * Add TaskEstimator docs
* Split channel resolver in two * Simplify WorkerResolverExtension and ChannelResolverExtension * Add default builder to ArrowFlightEndpoint * Add some docs * Listen to clippy * Split get_flight_client_for_url in two * Fix conflicts * Remove unnecessary channel resolver * Improve WorkerResolver docs * Use one ChannelResolver per runtime * Improve error reporting on client connection failure * Add a from_session_builder method for constructing an InMemoryChannelResolver * Add ChannelResolver and WorkerResolver default implementations for Arcs * Make TPC-DS tests use DataFusion test dataset * Remove non-working in-memory option from benchmarks * Remove unnecessary utils folder * Refactor benchmark folder * Rename to prepare_tpch.rs * Adapt benchmarks for TPC-DS * Update benchmarks README.md * Fix conflicts * Use default session state builder * Add ChildrenIsolatorUnionExec * Add proto serde for ChildrenIsolatorUnionExec * Wire up ChildrenIsolatorUnionExec to planner * Add integration tests for distributed UNIONs * Skip query 72 in TPC-DS benchmarks * Allow setting children isolator unions * Allow passing multiple queries in benchmarks * Extend TaskEstimator API to also be applicable to intermediate nodes
…b#277) * Support any dataset in benchmarks * Add TPC-DS schemas * Improve docs * Let Claude improve wording * Add custom_execution_plan.rs example * Link to custom_execution_plan.rs example * Respond to PR feedback
* Add ballista to benchmarks * Add Spark to benchmarks
* Doc fixes. * Minor additional changes.
* Allow passing any query id for benchmarks * Allow passing a custom query * Tidy up benchmarking code * Respond to PR feedbac
* Allow passing any query id for benchmarks * Allow passing a custom query * Tidy up benchmarking code * Allow passing a custom query * Introduce worker_connection_pool.rs for handing inter-worker connection batching * Allow passing any query id for benchmarks * Allow passing a custom query * Tidy up benchmarking code * Use ranges for the worker connection pool * Better debug messages
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.